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PCT/GB98/0080S 
GRAPHICAL VIDEO SYSTEMS 

The present invention relates to video display of 3-dimensionai 
graphical representations and more particularly to the provision of 
sporting games in a 3-D graphical format and viewing them in video 
from any desired viewpoint. 

Moreover, the 3-D graphical representation can be used for 
interactive participation of the viewer in a live sport event and to present 
to him stereo images of the game. 

A known problem with sporting events such as football matches is 
that storage and transmission of the video of such games requires an 
enormous amount of storage media and high bandwidth transmission 
links. It is normal to record such events on video tape and use high 
bandwidth RF or fibre optical links for transmission. Thus video 
recording and live transmission of sports events is very popular but has 
disadvantages. 

Firstly, when transmitted the full video requires substantially high 
bandwidth and is not convenient for real time transmission on low 
bandwidth media such as telephone lines or the internet. . 

Secondly, since video is a 2-dimensional representation that does 
not incorporate location information on objects, the viewpoint cannot be 
interactively manipulated by a viewer. 
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It is an object of the present invention to provide a record and live 
transmission of a match whilst requiring transmission and recording of 
substantial less data and whUst providing 3-D information on the objects. 
Hereinafter tliis will be described with reference to a soccer (football) 
match but it is to be understood that other events such as hockey, rugby 
football, American tboiball etc are to be included in this invention. Also 
sports such as sailing, motor racing, cycling etc in which the "players" 
are individually represented are also included within the definition of 
players. 

It is a further object of the present invention to provide an 
interactive system enabling manipulation of the match or event by the 
viewer. In particular it is a specific object to provide means for enabling 
a viewer to manipulate a game by viewing it from a desired angle, 
different from the original camera viewpoint at which the given video 
clip was shot. 

Over the Internet different viewers can share the same scene 
while communicating one with the other; or participating in the game 
during live transmission. 

This match manipulation can be done using recorded highlights of 
the game or can be done live during die time period that the game is 
being played. 

It is a further object of the present invention to allow for other 
forms of interactive manipulations like changing players and ball motion 
tracks o«^ gambling on play patterns. 
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The present invention provides a system for the video display of 
3-dimensional 3-D graphical representations of a live football game 
including a plurality of players said system comprising location detection 
means for detecting and providing an output representative of the 
location of each football player in each frame of. a video sequence, 
object storage means for storing 3-D graphical images of each football 
player and including combiner means for combining the output of the 
location detection means and Uie 3D graphical object storage means to 
provide a 3-D graphical video display representative of the plurality of 
football players in their correct position in each frame of the video 
sequence. 

Preferably die system also comprises manipulation means for 
enabling an operator to select a desired angle of view of the plurality of 
players. 

The present invention also provides means for displaying stereo 
images to the viewer and means to participate in the live game. 

The present invention also provides apparams for generating live 
remote video display of 3-dimensional graphical representations of a 
football match between first and second teams comprising : 

video camera means simated at a first location for videoing the 
football match played on a football pitch at the first location, 

player determination means simated at die first location for 
determining the position of at least the football players on the football 
pitch. 
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apparatus at a second location remote from said first location, 
said apparatus comprising means for storing graphical images of said 
football players, 

low bandwidth transmission link means for transmitting from the 
first location to the second location the positions of said football players, 

combination means situated at said second location for combining 
the location of each football player with a graphical image of the player, 
and 

display means for displaying at the second location a 3-D 
graphical representation of the football match. 

The apparatus can be provided with stereo-vision means enabling 
a viewer to view the game in 3-D vision. 

Preferably the apparatus ftiriher includes at said first location, or 
at a third location, remote from both said first and second locations 
player storage means storing graphical representations of the first and 
second team on said football pitch and including means for downloading 
said graphical representations of the first and second teams over said low 
bandwidth transmission link prior to commencement of the football 
match. 

Preferably the apparams at said second location comprises 
manipulation means operative by a viewer and enabling said viewer to 
select a different angle from which to view the football pitch from the 
angle of tlie pitch viewed by the video camera. 
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The apparatus may be provided with means for allowing a viewer 
to actively participate in a game. For example, in a motor racing event, 
tlie user can "enter" his own car and compete actively in a race against a 
real competitor. 

In a first embodiment the apparatus at the first location comprises 
means for measuring the camera field of view (FOV) and in which the 
player determination means comprises manual means for manually 
identifying the position of the players and a ball on the video image. 

In a second embodiment the apparatus at the first location 
includes pan/tilt/zoom and focus encoders for said video camera for 
automatic measurement of the FOV. 

In a third embodiment the apparanis at the first location includes 
cooperative means for identifying the location and/or identity of each 
player, the apparams including thermal imaging means for identification 
of the players position on the field. 

The thermal imaging means is preferably mounted on the camera 
and bore-sighted to the FOV of the camera. 

The cooperative means also preferably includes a laser scanner 
cooperating wiUi the thermal imaging means to scan each player and to 
receive from each player an identification signal generated by 
cooperative means on each player to identify each player on the pitch. 
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The low bandwidth transmission means preferably includes means 
for transmitting the players position and identity to the second location. 

The present invention also provides apparatus for generating a 
video display of 3-dimensional graphical representation of a football 
match between first and second teams including : 

video camera means siniated at a first location for videoing from 
a first angle tlie football match played on a football pitch at the first 
location, player determination means situated at said first location for 
determining the position of at least the football players viewed by said 
camera on the football pitch, 

including storage means for storing graphical images of the 
players, 

and including manipulation means for manipulating the video 
image produced by said video camera from said first angle to produce a 
video display of 3-dimension graphical representations of said football 
players from a second angle different to said first angle. 

Embodiments of the present invention will now be described, by 
way of example with reference to the accompanying drawings in which : 

Figure 1 shows schematically a football pitch with opposing 

teams; 

Figure 2 shows schematically a video camera overlooking a 
football pitch and Ulustrating the principles of die present invention; 

Figure 3 shows schematically the football pitch of Figure 1, 
togedier widi a video camera and illustrating various output signals 
useful injhree embodiments of tlie present invention; 
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Figure 4 shows block diagrammatical ly the circuitry for a first 
embodiment of the present invention utilising complete manual control 
and incorporating transmitter means for transmitting output information; 

Figure 5 shows block diagrammatically the circuitry for a second 
embodiment of die present invention incorporating a degree of manual 
control and incorporating transmitter means for transmission of an 
output signal including identification of the position and identity of 
players; 

Figure 6 shows block diagrammatically the circuitry for a diird 
embodiment of the present invention mcorporating automatic control and 
incorporating transmitter means for transmission of an output signal 
including identification of the position and identity of players; 

Figure 7 shows block diagrammatically simplified receiver 
circuitry for display and manipulation of die signals transmitted from the 
circuitry of Figures 5 and 6; and 

Figure 8 shows block diagrammatically an alternative 
embodiment for viewing a game using software generated for a 
commercially available game. 

The examples are described widi reference to a football match but 
as specified above this is only exemplary and this definition is intended 
to cover all televised sports events. Thus, references to grounds, ball 
and referee will apply to simUar objects in other sports such as puck, 
umpire and pitch etc. References to players can also refer to other 
objects such as boats, racing cars, horses etc. 

The present invention has for its principle invention a method and 
apparams for producing a 3-D graphical representation of a football 
match wfiich can be manipulated either after the event or live to provide 
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the viewer with a different view of the game to a TV camera situated in 
the football ground and in preferred embodiments to enable the remote 
viewer to select any desired angle to view the match. 

Three embodiments of the present will now be described firstly in 
general and then with reference to specific examples as shown in the 
drawings. 

QFF-LINF FOR MATCH HIGHTJGHTq 

The process steps tor off-line are as follows :- 

a) In this embodiment the camera(s) feed(s) is/are recorded 
on a VCR (Video Cassette Recorder). There is no 
requirement tor any additional equipment on the camera or 
players. 

b) The camera FOV (Field of View) and players/ball tracking 
is all done on the recorded video. These steps are not 
performed in real time bu t are done fast enough to have the 
game highlights just a short time after the match is over. 

This process can be accomplished using image processing 
techniques and video tracking methods as described in PCT Application 
No. WO 95/10915, the contents of which are incorporated herein by 
way of explanation. 

c) The camera, players and ball information is now used to 
locate computer generated models of the players and the 
ball on a synthetic playing field in the same manner as in 



8 



wo 98/46029 




PCT/GB98/0080S 



virtual Replay as described in the above mentioned PCT 
application. 

d) This now generates a 3-D representation. This means that 
the camera viewpoint can be dynamically changed 
throughout the video sequence, the "virtual" camera can 
"tly around" looking for the best angle to watch the play. 

e) The field by field updated information is minimal. 
Locations of objects, object identity, body direction etc. 
and this information can easily be transmitted in real time 
on the Plain Old Telephone System (POTS). 

f) Internet Application 

This application relates to a remote user with computer 
access to the Internet. 

Firstly, 3D models (stadium, players, ball etc) are loaded into the 
users computer. Team colours and texmres and players 
characteristics (black/white and possible face images, sky texture 
etc). According to the users request a given "graphical highlight" 
is now broadcast in real time to the users computer. In reality the 
camera and objects location is transmitted over the telephone line 
and the users computer locales die objects and renders the 3-D 
scene from a viewpoint which the user selects. 

g) Additionally, it is possible for different viewers to 
communicate with each odier and for such individual 
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viewers to participate in the game either alone or in 
competition witli each other, 
h) It is also possible by providing individual viewers with 3-D 
goggles to enable such viewers to obtain stereo vision 
views of a game. 

GRAPHICAL NON-rnO PERATTVF T TVE BROAnCA.ST 

a) In this embodiment the camera is equipped with 
pan/tilt/zoom/focus encoders on the camera mount and on 
tlie lens to measure the FOV direction and magnimde. 

b) the tracking of players and ball is done manually as 
explained hereinafter. 

GRAPHICAL COOPFRA TIVE TJVF. RRQADCAST 

a) In this embodiment manual intervention is eliminated by 
using a thermal imager; laser scanner and players units as 
described hereinafter. 



GENERAL PFSCRIPTION 

In order to derive the instantaneous locations in space of players 
and the ball, the following steps should be taken. 



t ^ « ^ • instantaneous direction and magnitude of the camera field of 

Y I measured or calculated. In the case of a moving camera 

( (crane) seven parameters should be measured, x,y,z,pan, tilt, roll 



^ « ^ I and zoom (focal length). In case of a fixed camera (most cameras 



>i , I »n football are stationary) only pan, tilt, roll and zoom are 
required. Roll is usually negligible, so we remain with pan, tilt 
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and zoom. These parameters can be measured in a number of 
ways : 

a) by attaching encoders (or other sensors) to the pan and tilt 
axes of the camera mount and on the zoom and focus 
motors; 

b) using pattern recognition and video tracking methods - 
assigning landmarks in the arena etc. 

Such systems are known from the above mentioned PCT 
application and also from US Patent No. 5, 543, 856 (Rosser) 
which is also incorporated herein by way of explanation. 

Objects on the field (players and ball) should be tracked (and 
identified) independently. This can also be done in several ways: 

a) using video tracking methods on the video recorded from 
the normal stadium cameras, as described in the above 
mentioned PCT application. The disadvantage is 
reliability. When the player is on a complex background, 
or when he is occluded by other players, the automatic 
tracking is not reliable and one needs manual override. 
This is why this method can be used only in post- 
production for the match highlights, 

b) real-time manual tracking of the players and ball using a 
calibrated theodolite or another pan/tilt unit with a 
projected reticle. The reticle is placed on the player and 
the player is tracked manually. A human tracker is 
required per each tracked object. Since each operator 
tracks just one object, identification is easy. 
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c) real-time automatic tracking and identification using a 
thermal imager boresighted on the stadium camera (since 
thermal contrasts are larger than the contrasts in the visible 
spectrum). For identification a laser scanner that "visits" 
each of the blobs detected by the thermal imager. A 
device is required on each player consisting of an electro- 
optical receiver and an RF transmitter that transmits the 
identity of the player to an RF receiver. 

d) combinations of b and c are possible, like automatic 
tracking of players, manual tracking the ball. 

When you have the camera's field of view direction and 
magnimde and tracking result of the players and the ball, it is still 
not enough because the tracking results are just directions. You 
now have to make the (correct) assumption that the p laying field 
is a plan^^Then the 3-D graphical representation of the football 
match can be created. 



With reference now to the drawings in Figure 1 , a football pitch 
10 is shown with opposing teams 0T1..0T3 and HT1...HT3 and a 
referee RF. Each player (including the referee) has a position on the 
pitch 10 and also a direction of movement indicated by arrow 12, as 
shown also in figure 2. 

The video camera 14 which is mounted to video the game is 
usually mounted, as shown in Figure 2 above, the pitch and "looks" 
down on the pitch. The computer program for the system assumes, 
correctly^in most instances, that the pitch is a plane surface 16. This 
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assumption enables tlie players to be correctly positioned on that plane 
surface 16. 

With reference now to Figure 3, in the three embodiments 
different outputs are required from the camera and other encoders. 

In die first embodiment only the camera video output CVOP is 
used, as explained widi reference to Figure 4. The camera output is fed 
to a VCR 400 where the game is recorded for subsequent display on a 
VDU 402. The game displayed on the VDU 402 is selectively sampled 
to select highlights showing specific interesting sequences. Once these 
have been selected the control unit 404 can be used to provide the 
necessary player and bail positioning information and player identity. 
The control unit 404 may be a mouse type arrangement or could be a 
touch sensitive screen on the VDU or a light pen arrangement. All that 
is necessary is to be able to track each player manually. Once a player 
has been identified, as explained above, the player can be tracked by 
known means, manual intervention only being required once a player 
moves out of the video sequence or is hidden by other players etc. 

For each sequence the position and identity of each player/ball is 
stored throughout the sequence in tracking store 406. 

In a graphics computer 408 such as, for example Silicon Graphics 
Server, the complete 3-D video sequence is composed by mixing the 
position/identity information with a 3-D graphic image stored in a store 
410. 
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The identity of each team - uniform, player identify etc can be 
input via input control 412 so that tlie completed data output from 
computer 408 comprises player position, identity and direction of 
movement of each player. This can be stored in store 414 and displayed 
for checking on VDU 416. 

Since the data stored in store 414 is 3-D graphic information the 
viewing angle can be adjusted using a control 418. This may be a 
joystick or other suitable control. 

The completed information can be transmitted 415 from store 414 
either with or without the graphic data relating to player identity as now 
explained with reference to Figures 5 and 6. 

In Figure 5, again the video output from camera 14 is fed to a 
VDU 502 via a video buffer store 500. The system is proposed to 
operate in real-time and is described with reference to a single player. 
For tracking of several players the system control may be duplicated. 

Each player is identified on a VDU 502 by a control 504 which as 
in Figure 4 may be a touch screen on the VDU. a mouse etc. This can 
be done in real-time for a single player. 

The information is stored in a manual tracking store 506. 

Each camera is provided with encoders 508 which give an 
encoder output EOP to store 510 where it is combined with the manual 
identity/position of each player. The output of control store 510 
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therefore comprises tlie identity and position of each player and then is 
transmitted by transmitter 512 to a viewer/internet user as described 
hereinafter with reference to Figure 7. The data transmitted may be 
readily transmitted on a low bandwidth medium such as a telephone line. 

In Figure 7 the transmitted data is received by any suitable 
receiving apparatus 700 and stored in a buffer store 702 and decoded, as 
necessary in decoder 704. In a video processor 706 the position/identity 
information is combined with 3-D graphical information stored in store 
708. This store contains graphical data relating to each team and 
possibly each player which has been downloaded prior to the live game. 
The output of the processor 706 can be displayed on a VDU 710 for the 
enjoyment of the viewer. Since the information in processor 706 is a 3- 
D graphical representation, the angle of view can be selected by the 
viewer using a control 712. 

With reference now to Figure 6, a more automated system is 
described. 

The camera output is still fed to a video circuit 600 and to a VDU 
602 for local viewing and for control functions such as quality control at 
the game. The video output is not however required in this embodiment 
for identification and location of the players. This is done using the 
camera encoders 508 (Figure 5) in conjunction with die output of a 
thermal imager 600 which is used to identify the position of each player 
on the pitch using the thermal output of each player. 
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Once a map of the location of each player has been identified the 
identity of each player Can be determined by using a laser scanner 602. 
This can scan tlie entire pitch or by use of information provided by 
imager 600 can be interrogated only when it scans a position where a 
player has been identified. Each player is provided with an optical/RF 
transponder which identifies each player. 

Thus into a location and identifier unit 604 the identity, and 
location, of each player is input and this data can be transmitted by 
transmitter 606 to the user/viewer as exemplified in Figure 7. 

The embodiments of Figure 6 requires no manual input and can 
transmit in real-time sufficient data to enable an internet user to generate 
a 3-D graphical representation of a football match (as defmed). The 
user/viewer can then select a desired angle of view. 

STEREO VLSTON 

The data transferred to each viewer can be used to provide the 
viewer with 3-D views. In the viewer side die 3-D information can be 
used to display 3 dimensional images using colour filter LCD shutter 
goggles 714 with first and second eye pieces 713,715. In diis case die 
computer of the viewer presents different images to each eye; for 
example to the right eye every even video field the computer draws the 
image as should be seen from die right eye and on every odd video field 
the computer draws the image corresponding to the left eye. The colour 
filter goggles allows Uie viewer's right eye to see only the even field and 
the left eye see only the odd fields, thus creating 3-D images. 



16 



wo 98/46029 




PCT/GB98/0080S 



INTERACTIVE LrVF-GAMF 

The system computer has all tlie information about the location of 
the real players in the field and transmits this information to the viewer 
location for presenting 3-D graphics. In die viewer side he can 
communicate widi other viewers via a transmitter 716 to share with them 
die same scene and his views. Moreover he can also participate in the 
game. For example, in Formula 1 racing die viewer receives die live 
transmission of the race in 3-D graphical representation on his screen 
He can have his own car provided by the graphics computer control of 
the control 712 and compete against die world champion drivers. The 3- 
D objects can have collision detectors, proximity detectors ... to comply 
widi the game rules. In a second example in a golf game die viewer can 
compete against one of the players. 

With reference to Figure 8, a modified embodiment is shown for 
a receiver circuitry enabling display of a real game using a commercially 
available 3D game software such as FIFA 98®, Nintendo® or a specially 
designed game software, all of which have die required 3D graphics to 
display a soccer game on the pitch. This software can be downloaded 
into a game software store 850 eidier from a CD unit 852 or from the 
internet via a suitable modem 854. 

In this embodiment the tracking system, as previously described, 
is used to extract the position of the players, ball and referee, the 
tracking system sends this data over a communication luik 800 to a 
central server 802 which transmits the data to all the clients connected to 
the server over the internet. 
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Each client uses a normal internet receiver connection to receive 
data from tiie internet, this may comprise a receiver 804 which can in 
normal manner supply usual internet data to a CPU 806 for displaying 
ona VDU 810. 

For the present invention the data received by receiver 804 is 
preferably stored in a store 812. The tracking data is then decoded in a 
decoder 814 and, if necessary, stored at least temporarily in a tracking 
data store 816. 

The tracking data is then converted by suitable interface software 
in a translator unit 818 into the format used by the game software. This 
translated output is then used via a control unit 820 to control the games 
software to display on a VDU 822 the actual movements in the real 
game by the 3D graphics of the "commercial" game. 

Since the software for control of the movement of players in the 
commercial 3D graphics game (such as FIFA 98®) is well understood in 
the art, the translation of tracking directions from the internet to conn-ol 
die movement of players in the commercial 3D game may be easily 
accomplished by persons skilled in the art of games software. 
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QhAlMfi 

I. A system for die video display of 3-dimensioiial (3-D) graphical 
representations of a live football game including a plurality of players 
said system comprising location detection means for detecting and 
providing an output representative of the location of each football player 
in each frame of a video sequence, object storage means for storing 3-D 
graphical images of each football player and including combiner means 
for combining the output of the location detection means and the 3-D 
graphical object storage means to provide a 3-D graphical video display 
representative of the plurality of football players in their correct position 
in each frame of the video sequence. 

2. A system as claimed in claim 1 further comprising manipulation 
means for enabling an operator to select a desired angle of view of the 
plurality of players. 

3. A system as claimed in claim 2 further comprising 3-D goggle 
means for enabling a viewer to view a scene in 3-D, said goggle means 
comprising first and second eyepieces and means for synchronising the 
first and second eyepieces to alternate video fields, each displaying a 
different view of the game, said different views corresponding to the 
views of a left and right eye of a spectator at said game to thereby 
provide said viewer with a 3-D view of the game. 

4. Apparatus for generating live remote video display of 3- 
dimensional graphical representations of a football match between first 
and second teams comprising : 
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video camera means situated at a first location for videoing the 
football match played oh a football pitch at the first location, 

player determination means situated at the first location for 
determining the position of at least the football players on the football 
pitch, 

apparams at a second location remote from said first location, 
said apparams comprising means for storing graphical images of said 
football players, 

low bandwidth transmission link means for transmitting from the 
first location to the second location the positions of said football players, 

combination means situated at said second locatfon for combining 
the location of each football player with a graphical image of the player, 
and 

display means for displaying at the second location a 3-D 
graphical representation of the football match. 

5. Apparatus as claimed in claim 4 in which the apparatus further 
includes at said first location, or at a third location, remote from both 
said first and second locations player storage means storing graphical 
representations of the first and second team on said football pitch and 
including means for downloading said graphical representations of the 
first and second teams over said low bandwidth transmission link prior 
to commencement of the football match. 

6. Apparams as claimed in claim 5 in which the apparatus at said 
second location comprises manipulation means operative by a viewer 
and enabling said viewer to select a different angle from which to view 
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the football pitch from tlie angle of the pitch viewed by the video 
camera. 

7. Apparatus as claimed in claim 6 including transmitter means 
enabling each viewer to communicate with other viewers to enable each 
viewer to compete with other viewers. 

8. Apparams as clauned in claim 6 or 7 including means for 
enabling a viewer to introduce further players from said graphics store 
and comprising means for controlling said further players to compete 
with the players transmitted from said game. 

9. Apparams as claimed in claim 8 in which said further players 
comprise racing cars. 



.10. Apparams as claimed in any one of claims 3 to 9 in which the 
apparatus at the first location comprises means for measuring die camera 
field of view (FOV) and in which the player determination means 
comprises manual means for manually identifying the position of the 
players and a ball on die video image. 

11. Apparatus as claimed in any one of claims 3 to 9 in which the 
apparams at the first location includes pan/tilt/zoom and focus encoders 
for said video camera for automatic measurement of the FOV. 

12. Apparams as claimed in any one of claims 3 to 9 in which the 
apparatus at the first location includes cooperative means for identifying 
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the location and/or identity of each player, the apparatus including 
thermal imaging means" for identification of the players position on the 
field. 

13. Apparatus as claimed in claim 12 in which the thermal imaging 
means is preferably mounted an the camera and bore-sighted to the FOV 
of the camera. 

14. Apparams as claimed in claim 12 or claim 13 in which the 
cooperative means also preferably includes a laser scanner cooperating 
with the thermal imaging means to scan each player and to receive from 
each player an identification signal generated by cooperative means on 
each player to identity each player on the pitch. 

15. Apparatus as claimed in any one of claims 4 to 14 in which the 
low bandwidth transmission means preferably includes means for 
transmitting the players position and identity to the second location. 

16. Apparatus for generating a video display of 3-dimensional 
graphical representation of a football match between first and second 
teams including : 

video camera means situated at a first location for videoing from 
a first angle the football match played on a football pitch at the fu-st 
location, player determination means situated at said first location for 
determining the position of at least the football players viewed by said 
camera on the football pitch, 

and including storage means for storing graphical unages of the 
players, - 
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and including manipulation means for manipulating the video 
image produced by said video camera from said first angle to produce a 
video display of 3-dimension graphical representations of said football 
players from a second angle different to said first angle. 
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